Skip to content

fix: Correct DynamoDB default time_format to include milliseconds#1530

Closed
claudespice wants to merge 25 commits into
trunkfrom
fix/dynamodb-time-format-default
Closed

fix: Correct DynamoDB default time_format to include milliseconds#1530
claudespice wants to merge 25 commits into
trunkfrom
fix/dynamodb-time-format-default

Conversation

@claudespice
Copy link
Copy Markdown
Collaborator

Summary

  • The DynamoDB connector docs showed the default time_format as 2006-01-02T15:04:05Z07:00 (without milliseconds)
  • The code defines the actual default as 2006-01-02T15:04:05.000Z07:00 (with .000 for millisecond precision)
  • This distinction matters because Go format strings are exact patterns — the .000 suffix is required to parse timestamps that include millisecond precision

Changes

  • Updated the default format pattern from 2006-01-02T15:04:05Z07:00 to 2006-01-02T15:04:05.000Z07:00 in the time format reference table
  • Fixed across vNext and versioned docs (1.9.x, 1.10.x, 1.11.x) — the DynamoDB docs don't exist in versions before 1.9.x

Reference

Verified against spiceai/spiceai at trunkcrates/runtime/src/dataconnector/dynamodb.rs (line 80: const DEFAULT_TIME_FORMAT: &str = "2006-01-02T15:04:05.000Z07:00")

Also verified at tag v1.9.2 — same default.

Claude added 4 commits April 18, 2026 03:11
…rsioned docs

Commit d5341c4 documented the Arrow nanosecond timestamp range
limitation (~1677–2262) for MSSQL and Oracle connectors, but only
in website/docs/ (vNext). The limitation existed in all prior versions
too — in versions 1.5.x–1.11.x, out-of-range timestamps silently
returned 1970-01-01 UTC instead of erroring.

Propagate the limitation note to all 7 versioned doc directories
(14 files total) with version-appropriate wording.
…d defaults

The docs listed pg_connection_pool_min_idle (default 1) and connection_pool_size
(default 5), but the code uses pg_connection_pool_min (default 5) and
connection_pool_size (default 10).
The docs stated the default ssl_mode is tls, but the code has always
used auto as the default since at least v1.5.0.
The docs showed the default time format as 2006-01-02T15:04:05Z07:00
but the code uses 2006-01-02T15:04:05.000Z07:00 (with .000 milliseconds).
This matters because Go format strings are exact patterns — without .000,
timestamps with millisecond precision would not parse correctly.
@claudespice claudespice added the bug Something isn't working label Apr 19, 2026
@lukekim lukekim self-assigned this Apr 21, 2026
@lukekim
Copy link
Copy Markdown
Contributor

lukekim commented Apr 21, 2026

@copilot resolve the merge conflicts in this pull request

Claude and others added 20 commits April 21, 2026 18:35
The code defines kafka_security_protocol with lowercase defaults and
options (sasl_ssl, plaintext, ssl, sasl_plaintext) in both the
ParameterSpec and runtime fallback, but the documentation showed
uppercase (SASL_SSL, PLAINTEXT, etc.).

Updated across all versioned docs (1.5.x through 1.11.x) and vNext.
Verified the code has used lowercase since at least v1.6.0.

Co-authored-by: lukekim <80174+lukekim@users.noreply.github.com>
Follow-up to #1507 which added scan_interval, ready_lag, endpoint_url,
and lag_exceeds_shard_retention_behavior to the vNext docs but missed
the versioned docs.

- version-1.11.x: all 4 parameters (verified against v1.11.5 code)
- version-1.10.x: 3 parameters — scan_interval, ready_lag, endpoint_url
  (lag_exceeds_shard_retention_behavior does not exist in v1.10.4)

Co-authored-by: lukekim <80174+lukekim@users.noreply.github.com>
Co-authored-by: lukekim <80174+lukekim@users.noreply.github.com>
The Datadog import-dashboard step read "Dashbord is now configured..."
across the unversioned docs and all 7 versioned copies. Addresses one of
the items called out in #506.

Co-authored-by: lukekim <80174+lukekim@users.noreply.github.com>
- Remove `duckdb_connection_string` which doesn't exist as a parameter
- Fix `open` → `duckdb_open` (component param, auto-prefixed)
- Note in-memory fallback behavior when param is omitted

Co-authored-by: lukekim <80174+lukekim@users.noreply.github.com>
The connector only supports username/password authentication via
dremio_username and dremio_password. Token/PAT authentication is not
implemented in the code — the only Credentials variant used is
UsernamePassword.

Co-authored-by: lukekim <80174+lukekim@users.noreply.github.com>
- Remove nonexistent `graphql_endpoint` (endpoint comes from `from:` path)
- Remove nonexistent `graphql_pagination_parameters` (pagination is automatic)
- Fix `graphql_json_pointer` → `json_pointer` (runtime param, no prefix)
- Add missing `graphql_auth_token` parameter
- Fix `graphql_auth_header` description to match actual behavior

Co-authored-by: lukekim <80174+lukekim@users.noreply.github.com>
…uth notes (#1536)

Co-authored-by: lukekim <80174+lukekim@users.noreply.github.com>
…1535)

Co-authored-by: lukekim <80174+lukekim@users.noreply.github.com>
Co-authored-by: lukekim <80174+lukekim@users.noreply.github.com>
)

* fix: Add nanosecond timestamp range limitation to MSSQL and Oracle versioned docs

Commit d5341c4 documented the Arrow nanosecond timestamp range
limitation (~1677–2262) for MSSQL and Oracle connectors, but only
in website/docs/ (vNext). The limitation existed in all prior versions
too — in versions 1.5.x–1.11.x, out-of-range timestamps silently
returned 1970-01-01 UTC instead of erroring.

Propagate the limitation note to all 7 versioned doc directories
(14 files total) with version-appropriate wording.

* fix: Correct PostgreSQL accelerator connection pool parameter name and defaults

The docs listed pg_connection_pool_min_idle (default 1) and connection_pool_size
(default 5), but the code uses pg_connection_pool_min (default 5) and
connection_pool_size (default 10).

* fix: Correct IMAP ssl_mode default from tls to auto

The docs stated the default ssl_mode is tls, but the code has always
used auto as the default since at least v1.5.0.

* fix: Correct DynamoDB default time_format to include milliseconds

The docs showed the default time format as 2006-01-02T15:04:05Z07:00
but the code uses 2006-01-02T15:04:05.000Z07:00 (with .000 milliseconds).
This matters because Go format strings are exact patterns — without .000,
timestamps with millisecond precision would not parse correctly.

* fix: Document PostgreSQL replication parameters for WAL streaming

The PostgreSQL connector added logical replication (WAL streaming) support
in PR #10364 with 5 new parameters (pg_replication_slot, pg_publication,
pg_replication_initial_snapshot, pg_replication_temporary_slot,
pg_replication_status_interval) that were not yet documented.

---------

Co-authored-by: Claude <claude@Claudes-Mini.localdomain>
Co-authored-by: lukekim <80174+lukekim@users.noreply.github.com>
…C option (#1541)

* faq: add incremental ingestion Q and native PostgreSQL replication CDC option

- Add new FAQ entry pointing to data-refresh docs for incremental ingestion
- Extend CDC FAQ entry with native PostgreSQL logical replication (recommended for PG)
- Add new Features > CDC > PostgreSQL Logical Replication page documenting
  wal_level=logical + pgoutput direct replication (no Debezium/Kafka)
- Update CDC overview to list native PG replication alongside Debezium

* faq, cdc: mention direct Kafka and DynamoDB Streams ingestion

* docs: fix broken links by using .md extensions for cross-page refs

* data-refresh: add combined incremental ingestion example (overlap, retention, soft deletes)
Co-authored-by: lukekim <80174+lukekim@users.noreply.github.com>
… DataDog OTLP setup (#1542)

Co-authored-by: lukekim <80174+lukekim@users.noreply.github.com>
The scylladb_ssl parameter is defined in the ParameterSpec but
never read or applied in the session builder code. The parameter
is accepted without error but has no effect on the connection.

Added a note to the parameter description in both vNext and
1.11.x docs.

Co-authored-by: lukekim <80174+lukekim@users.noreply.github.com>
The documentation incorrectly states that the default value of
pg_sslmode is verify-full. The actual default is prefer, which
matches standard libpq/tokio-postgres behavior — when no sslmode
is specified, the client tries TLS but falls back to plaintext.

For the replication (CDC) path, SslMode::from_str_or_default()
explicitly defaults to Prefer (postgres_replication/config.rs).
For the read path, no sslmode default is set in the ParameterSpec,
so the underlying tokio-postgres library applies its own default
of prefer.

Fixed across all versioned docs (1.5.x–1.11.x) and vNext.

Co-authored-by: lukekim <80174+lukekim@users.noreply.github.com>
Add a new Multi-Vector Search feature page covering column-of-vectors
embeddings (List<Utf8> source columns), aggregation strategies
(max/mean/sum), max_elements_per_row caps, and ColBERT-style
late-interaction multi-query vector_search.

Cross-link from the search index, vector search, and embeddings
component pages, and document the previously-missing aggregation and
max_elements_per_row fields in the datasets reference.

Co-authored-by: lukekim <80174+lukekim@users.noreply.github.com>
Co-authored-by: lukekim <80174+lukekim@users.noreply.github.com>
Co-authored-by: lukekim <80174+lukekim@users.noreply.github.com>
Co-authored-by: lukekim <80174+lukekim@users.noreply.github.com>
…d defaults (#1528)

* fix: Add nanosecond timestamp range limitation to MSSQL and Oracle versioned docs

Commit d5341c4 documented the Arrow nanosecond timestamp range
limitation (~1677–2262) for MSSQL and Oracle connectors, but only
in website/docs/ (vNext). The limitation existed in all prior versions
too — in versions 1.5.x–1.11.x, out-of-range timestamps silently
returned 1970-01-01 UTC instead of erroring.

Propagate the limitation note to all 7 versioned doc directories
(14 files total) with version-appropriate wording.

* fix: Correct PostgreSQL accelerator connection pool parameter name and defaults

The docs listed pg_connection_pool_min_idle (default 1) and connection_pool_size
(default 5), but the code uses pg_connection_pool_min (default 5) and
connection_pool_size (default 10).

---------

Co-authored-by: Claude <claude@Claudes-Mini.localdomain>
Co-authored-by: lukekim <80174+lukekim@users.noreply.github.com>
Co-authored-by: lukekim <80174+lukekim@users.noreply.github.com>
Copy link
Copy Markdown

Copilot AI commented Apr 21, 2026

@copilot resolve the merge conflicts in this pull request

Resolved. The branch was rebased and now contains only the DynamoDB time_format fix (4 files) relative to trunk — the PostgreSQL, IMAP, and MSSQL/Oracle changes that were bundled in the original commits had already been merged into trunk separately. Commit 2fe576e cleaned up the stale conflicts.

Copilot AI requested a review from lukekim April 21, 2026 18:40
@lukekim lukekim closed this Apr 27, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/docs bug Something isn't working release

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants